Test Utility literally refers to using a test as a utility, as a tool. And in our context we mean using tests as a toal for selection.
The psychometric properties of the test determine the relationship between costs and benefits.
These properties can be empirically tested. We will show you the reasoning process in this lecture.
Often we want to predict based on test scores how an individual will perform at a later time.
The predictive validity is defined as the correlation between the predictor / test score and the criterion.
NOTE: This is no different than the \(R\) in regular regression analysis; the correlation between the outcome and the model.
With a correlation of zero,
selection is useless
Based on our criterion and predictor we need to determine:
- When we trully know someone is competent.
- At what predictor score we want to select someone.
That means setting a:
- Cut-off for the criterion (Base Rate)
- Cut-off for the predictor (Selection Ratio)
| FN | TP | 0 |
| TN | FP | 0 |
| 0 | 0 | 0 |
What is the:
Admission test (NL: selectie aan de poort)
|
FN 102 |
TP 162 |
|
TN 163 |
FP 85 |
Percentage of correct conclusions.
\[\definecolor{green}{RGB}{0,128,0} \text{Hit Rate} = \frac{{\color{green}\text{TP}} + \color{green}\text{TN}}{N}\]
\[\frac{162 + 163}{512} = 0.63\]
|
FN 102 |
TP 162 |
|
TN 163 |
FP 85 |
Percentage of correct conclusions when not selecting.
\[\definecolor{red}{RGB}{255,0,0} \text{Base Rate} = \frac{{\color{green}\text{TP}} + \color{red}\text{FN}}{N}\]
\[\frac{162 + 102}{512} = 0.52\]
To determine the benefit of selection, we need to know what the hit rate is compared to the base rate.
\[\text{Hit Rate} - \text{Base Rate} = \text{Benefit}\]
\[0.63 - 0.52 = 0.11\]
|
FN 102 |
TP 162 |
|
TN 163 |
FP 85 |
We can determine the efficiency of selection by calculating the sensitivity and specificity of a test.
Sensitivity: Percentage of eligible candidates that will be selected.
\[\text{Sensitivity} = \frac{\color{green}\text{TP}}{{\color{red}\text{FN}} + \color{green}\text{TP}}\]
\[\frac{162}{102 + 162} = 0.61\]
The lower the sensitivity
The more frustrated
parents and students
|
FN 102 |
TP 162 |
|
TN 163 |
FP 85 |
Specificity: Percentage of inapt candidates that will be rejected.
\[\text{Specificity} = \frac{\color{green}\text{TN}}{{\color{green}\text{TN}} + \color{red}\text{FP}}\]
\[\frac{163}{163 + 85} = 0.66\]
Only if you select everyone
you can determine
the quality of the procedure
So, what if you do not have the resources to conduct empirical research?
In that case, you can resort to using Taylor-Russell tables, hhich provide estimates of the percentage of eligible candidates of those selected.
The two factors influencing the cell values “The proportion satisfactory among those selected” are:
So, there are two ways to increase efficiency
Criterion validity is threatened by all factors that reduce the relation between predictor and criterion.
Criterion validity can be increased by all measures that increase the correlation.
Just wanting selection
to work does not
influence the quality
of selection